1. Field of the Invention
The present invention relates to an image processing apparatus, an image capturing apparatus, an image processing method, and a storage medium for obtaining information on focus control of a subject.
2. Description of the Related Art
As an image capturing apparatus, such as a digital camera, there is known an apparatus that makes focus adjustment of a subject included in a captured image by a user specifying the subject by a touch panel operation etc. (for example, Japanese Patent Laid-Open No. 2011-39457).
Further, the technique is also proposed, which makes it possible for a user to easily grasp an object in focus the image of which is to be captured (subject) by displaying a map image for indicating a depth position representing a position of the object the image of which is to be captured in the depth direction and a focus position representing the position of focus (for example, Japanese Patent Laid-Open No. 2010-177741).
In recent years, the light field photography technique is also known, which acquires information of the orientation and intensity of a light ray (light field data) and makes adjustment of the focus position (refocus) and adjustment of the depth of field by the later image processing (for example, Japanese Patent Laid-Open No. 2008-515110).
By using this technique, there is an advantage that a failure in focus adjustment at the time of image capturing can be made up by image processing because it is possible to make focus adjustment after image capturing. Further, there is also an advantage that it is possible to obtain a plurality of images in which arbitrary subjects are brought into focus in an image from one captured image by changing the image processing method, and therefore, it is possible to reduce the number of times of image capturing.
In light field photography, from data of images captured from a plurality of viewpoints, the direction and intensity of a light ray that passes through each position (light field) in a plurality of positions in the space are calculated. Then, by using the information of the obtained light field, an image on the assumption that light passes through a virtual optical system and forms the image on a virtual sensor is calculated. By appropriately setting such a virtual optical system and a virtual sensor, refocus described previously is enabled. As an image capturing apparatus for acquiring the light field, a plenoptic camera (for example, Japanese Patent Laid-Open No. 2009-124213) in which a microlens array is arranged behind a main lens and a camera array in which compact cameras are arranged side by side are known. It is possible for both to obtain data of subject images from a plurality of viewpoints captured from different directions by one-time image capturing. In other words, it is possible to represent the light field photography as calculation of an image to be acquired by a virtual sensor under virtual optical conditions from data of images from a plurality of viewpoints. In the following, the processing to calculate an image acquired by the virtual sensor is referred to as “refocus processing”. As refocus processing, there is known a method in which acquired data of images from a plurality of viewpoints is subjected to projective transformation onto a virtual sensor, and added and averaged (for example, WO2008050904).
As a method for displaying an image having been subjected to refocus processing (hereinafter, a refocus image) while changing the focus position, mention is made of, for example, the method disclosed in United States Patent No. 2008/0131019. In the method disclosed in United States Patent No. 2008/0131019, the user interface (UI) to adjust the focus position is prepared on the screen on which a refocus image is displayed and the focus position is changed via the UI. Further, in United States Patent No. 2008/0131019, the method is disclosed, which displays a refocus image whose focus position is adjusted to a subject which a user desires to put to a focus position and has given instructions through a screen on which the refocus image is displayed.
In the display of captured image data or light field data, an object to be brought into focus exists on a plane at a fixed distance from the position of image capturing. However, in the case where focusing is performed by specifying a specific subject, it is not possible for a user to grasp which subject other than the specific subject is brought into focus.
Further, by the technique disclosed in Japanese Patent Laid-Open No. 2010-177741, it is possible to grasp which subject is brought into focus by the map image in which marks indicating objects the image of which is to be captured are arranged in order in the depth direction, however, the map image is quite different from the captured image (actual image), and therefore, it is difficult to grasp the subject intuitively.
That is, by the conventional methods disclosed in the above-described Patent Literatures etc., on a subject in which range, refocus can be performed is not specified clearly at the time of photographing or at the time of editing an image, and therefore, there is such a problem that it is difficult for a user to perform image capturing or image edition in a manner the user intends.